Everything is executed on single node, average of 10 runs.

Metrics

Some of reported HPCC metrics:

Results

Overall Performance (That is per All MPI Processes)

High Performance LINPACK Floating-Point Performance, MFLOP per Second

High Performance LINPACK Floating-Point Performance, MFLOP per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 46535.57 237.15
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 47026.24 141.35
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 45474.51 69.34
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 581802.00 3172.57
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 435367.30 991.71
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 330640.90 2516.47
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 485480.90 1243.16
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 458037.20 2048.63
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 421653.10 3126.48
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 988258.60 20198.84
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 987046.80 14078.96
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 515101.60 10552.39
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 554204.50 62818.10

Fast Fourier Transform (FFTW) Floating-Point Performance, MFLOP per Second

Fast Fourier Transform (FFTW) Floating-Point Performance, MFLOP per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 15388.25 189.37
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 16445.68 513.88
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 18292.79 981.23
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 15490.95 340.47
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 16954.09 226.83
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 17948.42 1564.26
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 15247.34 165.05
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 15983.20 422.39
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 17292.98 4492.64
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 43670.26 780.33
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 46666.67 502.63
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 21765.33 1191.43
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 14499.70 304.43

Parallel Matrix Transpose (PTRANS), MByte per Second

Parallel Matrix Transpose (PTRANS), MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 10720.32 326.78
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 9841.70 136.43
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 16847.96 2121.25
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 10654.13 279.48
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 10037.08 191.98
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 18119.24 1325.63
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 10576.29 438.45
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 9855.29 260.41
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 17883.64 1688.42
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 18074.71 446.00
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 13965.83 209.40
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 9624.68 146.03
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 7165.57 173.95

MPI Random Access, MUpdate per Second

MPI Random Access, MUpdate per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 259.84 3.36
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 252.05 4.77
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 173.59 6.85
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 264.26 2.07
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 248.71 5.50
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 177.20 14.11
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 261.14 2.88
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 253.37 3.62
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 176.40 14.10
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 39.27 0.50
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 3.26 0.03
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 40.03 0.99
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 118.39 18.94

Average Double-Precision General Matrix Multiplication (DGEMM) Floating-Point Performance, MFLOP per Second

Average Double-Precision General Matrix Multiplication (DGEMM) Floating-Point Performance, MFLOP per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 57371.47 66.41
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 61774.21 64.72
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 64949.94 178.61
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 777500.16 2118.86
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 523424.00 4498.68
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 452975.36 3563.49
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 630826.24 3461.07
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 517666.05 1939.81
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 414587.39 3954.54
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 2106614.40 77528.00
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 334587.27 14445.84
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 696314.16 13099.94
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 724319.12 13386.87

Average STREAM ‘Triad’ Memory Bandwidth, MByte per Second

Average STREAM ‘Triad’ Memory Bandwidth, MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 157604.12 1781.38
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 170510.52 9350.40
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 173858.07 56420.11
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 158843.60 2085.14
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 167578.30 9121.34
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 154225.42 20570.22
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 157605.30 2097.07
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 169241.35 6080.71
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 147068.76 1834.26
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 149741.62 2748.06
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 315214.94 23818.48
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 89959.69 872.67
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 87356.19 532.06

Average STREAM ‘Add’ Memory Bandwidth, MByte per Second

Average STREAM ‘Add’ Memory Bandwidth, MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 151023.98 1806.49
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 169686.86 3973.33
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 179711.27 68912.78
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 150678.80 1669.13
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 173855.87 6920.44
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 169258.83 32883.04
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 152415.90 7146.12
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 170028.17 8595.58
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 155504.82 2356.32
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 149137.05 2318.96
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 316756.66 21596.90
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 89668.49 981.04
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 86956.61 489.94

Average STREAM ‘Copy’ Memory Bandwidth, MByte per Second

Average STREAM ‘Copy’ Memory Bandwidth, MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 129057.8 5813.76
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 143716.5 5157.72
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 163230.1 65730.92
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 132548.7 6335.55
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 137553.0 1679.58
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 168196.8 37854.83
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 133553.3 6625.87
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 139392.5 1139.72
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 146985.1 12762.55
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 132971.1 2353.97
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 256390.0 7269.67
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 95779.5 306.79
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 100068.5 385.94

Average STREAM ‘Scale’ Memory Bandwidth, MByte per Second

Average STREAM ‘Scale’ Memory Bandwidth, MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 145269.72 3584.16
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 151182.25 4360.15
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 152769.40 50916.58
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 143432.81 923.46
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 149246.44 4300.89
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 144922.80 24866.73
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 146640.08 2347.68
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 145871.47 6602.79
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 138834.45 2532.77
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 131272.56 2988.52
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 270395.12 20602.43
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 79454.04 361.65
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 77124.61 480.41

Performance per Core (that is MPI Process)

High Performance LINPACK Floating-Point Performance, MFLOP per Second

High Performance LINPACK Floating-Point Performance, MFLOP per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 727.12 237.15
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 367.39 141.35
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 177.63 69.34
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 9090.66 3172.57
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 3401.31 991.71
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 1291.57 2516.47
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 7585.64 1243.16
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 3578.42 2048.63
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 1647.08 3126.48
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 20588.72 20198.84
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 14515.39 14078.96
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 21462.57 10552.39
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 19793.02 62818.10

Fast Fourier Transform (FFTW) Floating-Point Performance, MFLOP per Second

Fast Fourier Transform (FFTW) Floating-Point Performance, MFLOP per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 240.44 189.37
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 128.48 513.88
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 71.46 981.23
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 242.05 340.47
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 132.45 226.83
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 70.11 1564.26
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 238.24 165.05
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 124.87 422.39
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 67.55 4492.64
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 909.80 780.33
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 686.27 502.63
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 906.89 1191.43
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 517.85 304.43

Parallel Matrix Transpose (PTRANS), MByte per Second

Parallel Matrix Transpose (PTRANS), MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 167.50 326.78
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 76.89 136.43
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 65.81 2121.25
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 166.47 279.48
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 78.41 191.98
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 70.78 1325.63
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 165.25 438.45
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 76.99 260.41
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 69.86 1688.42
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 376.56 446.00
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 205.38 209.40
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 401.03 146.03
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 255.91 173.95

MPI Random Access, MUpdate per Second

MPI Random Access, MUpdate per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 4.06 3.36
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 1.97 4.77
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 0.68 6.85
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 4.13 2.07
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 1.94 5.50
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 0.69 14.11
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 4.08 2.88
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 1.98 3.62
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 0.69 14.10
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 0.82 0.50
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 0.05 0.03
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 1.67 0.99
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 4.23 18.94

Average Double-Precision General Matrix Multiplication (DGEMM) Floating-Point Performance, MFLOP per Second

Average Double-Precision General Matrix Multiplication (DGEMM) Floating-Point Performance, MFLOP per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 896.43 66.41
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 482.61 64.72
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 253.71 178.61
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 12148.44 2118.86
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 4089.25 4498.68
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 1769.43 3563.49
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 9856.66 3461.07
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 4044.27 1939.81
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 1619.48 3954.54
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 43887.80 77528.00
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 4920.40 14445.84
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 29013.09 13099.94
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 25868.54 13386.87

Average STREAM ‘Triad’ Memory Bandwidth, MByte per Second

Average STREAM ‘Triad’ Memory Bandwidth, MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 2462.56 1781.38
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 1332.11 9350.40
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 679.13 56420.11
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 2481.93 2085.14
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 1309.21 9121.34
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 602.44 20570.22
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 2462.58 2097.07
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 1322.20 6080.71
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 574.49 1834.26
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 3119.62 2748.06
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 4635.51 23818.48
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 3748.32 872.67
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 3119.86 532.06

Average STREAM ‘Add’ Memory Bandwidth, MByte per Second

Average STREAM ‘Add’ Memory Bandwidth, MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 2359.75 1806.49
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 1325.68 3973.33
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 702.00 68912.78
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 2354.36 1669.13
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 1358.25 6920.44
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 661.17 32883.04
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 2381.50 7146.12
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 1328.35 8595.58
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 607.44 2356.32
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 3107.02 2318.96
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 4658.19 21596.90
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 3736.19 981.04
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 3105.59 489.94

Average STREAM ‘Copy’ Memory Bandwidth, MByte per Second

Average STREAM ‘Copy’ Memory Bandwidth, MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 2016.53 5813.76
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 1122.79 5157.72
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 637.62 65730.92
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 2071.07 6335.55
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 1074.63 1679.58
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 657.02 37854.83
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 2086.77 6625.87
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 1089.00 1139.72
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 574.16 12762.55
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 2770.23 2353.97
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 3770.44 7269.67
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 3990.81 306.79
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 3573.87 385.94

Average STREAM ‘Scale’ Memory Bandwidth, MByte per Second

Average STREAM ‘Scale’ Memory Bandwidth, MByte per Second
resource exe_type cpus average stdev
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 64 2269.84 3584.16
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 128 1181.11 4360.15
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/sysblas/intfft 256 596.76 50916.58
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 64 2241.14 923.46
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 128 1165.99 4300.89
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/openblas/intfft 256 566.10 24866.73
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 64 2291.25 2347.68
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 128 1139.62 6602.79
Koinu (CPU ARM ThunderX2 32(128) cores x2) openmpi/armpl/intfft 256 542.32 2532.77
Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) impi/mkl 48 2734.84 2988.52
Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) impi/mkl 68 3976.40 20602.43
Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) mvapich2/mkl 24 3310.58 361.65
Bridges (Intel E5-2695 v3 14 cores x2, Haswell) impi/mkl 28 2754.45 480.41
LS0tCnRpdGxlOiAiSFBDQyIKYXV0aG9yOiBuaWtvbGF5cwpkYXRlOiA3LzI4LzIwCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUpCgpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkoZHBseXIpCmxpYnJhcnkoY293cGxvdCkKbGlicmFyeShrbml0cikKbGlicmFyeShrYWJsZUV4dHJhKQpgYGAKCmBgYHtyIGluY2x1ZGU9RkFMU0V9CiMgUmVhZCBkYXRhCmhwY2MgPC0gcmVhZC5jc3YoImRhdGEvaHBjYy5jc3YiLCBzdHJpbmdzQXNGYWN0b3JzPUYpICU+JQogICAgc2VsZWN0KC1YKSAlPiUgZmlsdGVyKG1ldHJpYyAhPSAiQXBwOlZlcnNpb24iKSAlPiUKICAgIG11dGF0ZSgKICAgICAgICB2YWx1ZT1hcy5udW1lcmljKHZhbHVlKSwKICAgICAgICBleGVfdHlwZT1mYWN0b3IoZXhlX3R5cGUpLAogICAgICAgIHByb2JsZW1fc2l6ZT1mYWN0b3IocHJvYmxlbV9zaXplKSwKICAgICAgICBtZXRyaWNfdHlwZT1mYWN0b3IobWV0cmljX3R5cGUpLAogICAgICAgIG1ldHJpYz1mYWN0b3IobWV0cmljKSwKICAgICAgICB1bml0cz1mYWN0b3IodW5pdHMpLAogICAgICAgIHJlc291cmNlPWZhY3RvcihyZXNvdXJjZSwgYygKICAgICAgICAgICAgIktvaW51IChDUFUgQVJNIFRodW5kZXJYMiAzMigxMjgpIGNvcmVzIHgyKSIsCiAgICAgICAgICAgICJTdGFtcGVkZTIgU0tYIChDUFUgSW50ZWwgWGVvbiBQbGF0aW51bSA4MTYwIDI0KDQ4KSBjb3JlcyB4MikiLAogICAgICAgICAgICAiU3RhbXBlZGUyIEtOTCAoQ1BVIEludGVsIFhlb24gUGhpIDcyNTAgNjgoMjcyKSBjb3JlcyB4MikiLAogICAgICAgICAgICAiQ29tZXQgKENQVSBJbnRlbCBYZW9uIEU1LTI2ODB2MyAxMiBjb3JlcyB4MiwgSGFzd2VsbCkiLAogICAgICAgICAgICAiQnJpZGdlcyAoSW50ZWwgRTUtMjY5NSB2MyAxNCBjb3JlcyB4MiwgSGFzd2VsbCkiKSksCiAgICAgICAgZXhlX3R5cGU9ZmFjdG9yKGV4ZV90eXBlLCBjKAogICAgICAgICAgICAib3Blbm1waS9zeXNibGFzL2ludGZmdCIsCiAgICAgICAgICAgICJvcGVubXBpL29wZW5ibGFzL2ludGZmdCIsCiAgICAgICAgICAgICJvcGVubXBpL2FybXBsL2ludGZmdCIsCiAgICAgICAgICAgICJpbXBpL21rbCIsICJtdmFwaWNoMi9ta2wiKSkKICAgICkKI2RwdXQodW5pcXVlKGhwY2MkZXhlX3R5cGUpKQojaGVhZChocGNjKQpgYGAKCmBgYHtyIGluY2x1ZGU9RkFMU0V9CiMgc3RhdGlzdGljcyB3ZSBhcmUgaW50ZXJlc3RlZCBpbgojZHB1dCh1bmlxdWUoaHBjYyRtZXRyaWNbaHBjYyRtZXRyaWNfdHlwZT09InN0YXRpc3RpYyJdKSkKc3RhdHNfb3ZlcmFsbCA8LSBjKAogICAgIkhpZ2ggUGVyZm9ybWFuY2UgTElOUEFDSyBGbG9hdGluZy1Qb2ludCBQZXJmb3JtYW5jZSIsCiAgICAiRmFzdCBGb3VyaWVyIFRyYW5zZm9ybSAoRkZUVykgRmxvYXRpbmctUG9pbnQgUGVyZm9ybWFuY2UiLAogICAgIlBhcmFsbGVsIE1hdHJpeCBUcmFuc3Bvc2UgKFBUUkFOUykiLAogICAgIk1QSSBSYW5kb20gQWNjZXNzIgopCnN0YXRzX3Blcl9jb3JlIDwtIGMoCiAgICAiQXZlcmFnZSBEb3VibGUtUHJlY2lzaW9uIEdlbmVyYWwgTWF0cml4IE11bHRpcGxpY2F0aW9uIChER0VNTSkgRmxvYXRpbmctUG9pbnQgUGVyZm9ybWFuY2UiLAogICAgIkF2ZXJhZ2UgU1RSRUFNICdUcmlhZCcgTWVtb3J5IEJhbmR3aWR0aCIsCiAgICAiQXZlcmFnZSBTVFJFQU0gJ0FkZCcgTWVtb3J5IEJhbmR3aWR0aCIsCiAgICAiQXZlcmFnZSBTVFJFQU0gJ0NvcHknIE1lbW9yeSBCYW5kd2lkdGgiLAogICAgIkF2ZXJhZ2UgU1RSRUFNICdTY2FsZScgTWVtb3J5IEJhbmR3aWR0aCIKKQp1bml0cyA8LSBocGNjICU+JSBncm91cF9ieShtZXRyaWMpICU+JSBzdW1tYXJpc2UodW5pdHM9YXMuY2hhcmFjdGVyKGZpcnN0KHVuaXRzKSkpICU+JQogICAgdW5ncm91cCgpCnVuaXRzJHVuaXRzW2lzLm5hKHVuaXRzJHVuaXRzKV08LSIiCmBgYAoKRXZlcnl0aGluZyBpcyBleGVjdXRlZCBvbiBzaW5nbGUgbm9kZSwgYXZlcmFnZSBvZiAxMCBydW5zLgoKIyMgTWV0cmljcwoKU29tZSBvZiByZXBvcnRlZCBIUENDIG1ldHJpY3M6CgogKiBBdmVyYWdlIERvdWJsZS1QcmVjaXNpb24gR2VuZXJhbCBNYXRyaXggTXVsdGlwbGljYXRpb24gKERHRU1NKSBGbG9hdGluZy1Qb2ludCBQZXJmb3JtYW5jZQogICArIEVtYmFycmFzc2luZ2x5IHBhcmFsbGVsLCBBdmVyYWdlZCBvdmVyIGFsbCBNUEkgcHJvY2VzcywgTUZMT1AgcGVyIFNlY29uZCBwZXIgTVBJIHByb2Nlc3MKICAgKyAoVGhlcmUgaXMgYWxzbyBzZXJpYWwgbWV0cmljOyBtZWFzdXJlZCBidXQgbm90IGFuYWx5c2VkKQogKiBIaWdoIFBlcmZvcm1hbmNlIExJTlBBQ0sgRmxvYXRpbmctUG9pbnQgUGVyZm9ybWFuY2UKICAgKyBQYXJhbGxlbCwgb3ZlcmFsbCBNRkxPUCBwZXIgU2Vjb25kCiAqIEZhc3QgRm91cmllciBUcmFuc2Zvcm0gKEZGVFcpIEZsb2F0aW5nLVBvaW50IFBlcmZvcm1hbmNlCiAgICsgUGFyYWxsZWwsIE1GTE9QIHBlciBTZWNvbmQKICAgKyAoVGhlcmUgYXJlIGFsc28gc2VyaWFsIGFuZCBlbWJhcnJhc3NpbmdseSBwYXJhbGxlbCBtZXRyaWNzOyBtZWFzdXJlZCBidXQgbm90IGFuYWx5c2VkKQogKiBQYXJhbGxlbCBNYXRyaXggVHJhbnNwb3NlIChQVFJBTlMpCiAgICsgUGFyYWxsZWwsIG92ZXJhbGwgTUJ5dGUgcGVyIFNlY29uZAogKiBNUEkgUmFuZG9tIEFjY2VzcwogICArIFBhcmFsbGVsLCBvdmVyYWxsIE1VcGRhdGUgcGVyIFNlY29uZAogICArIChUaGVyZSBhcmUgYWxzbyBzZXJpYWwgYW5kIGVtYmFycmFzc2luZ2x5IHBhcmFsbGVsIG1ldHJpY3M7IG1lYXN1cmVkIGJ1dCBub3QgYW5hbHlzZWQpCiAqIEF2ZXJhZ2UgU1RSRUFNICdUcmlhZCcvICdBZGQnIC8gJ0NvcHknIC8gJ1NjYWxlJyBNZW1vcnkgQmFuZHdpZHRoCiAgICsgRW1iYXJyYXNzaW5nbHkgcGFyYWxsZWwsIEF2ZXJhZ2VkIG92ZXIgYWxsIE1QSSBwcm9jZXNzLCBNQnl0ZSBwZXIgU2Vjb25kCiAgICsgKFRoZXJlIGlzIGFsc28gc2VyaWFsIG1ldHJpYzsgbWVhc3VyZWQgYnV0IG5vdCBhbmFseXNlZCkKCiMjIFJlc3VsdHMKIyMjIE92ZXJhbGwgUGVyZm9ybWFuY2UgKFRoYXQgaXMgcGVyIEFsbCBNUEkgUHJvY2Vzc2VzKQoKYGBge3IgcmVzdWx0cz0nYXNpcycsIGVjaG89RkFMU0UsIGZpZy53aWR0aCA9IDExLCBmaWcuaGVpZ2h0ID0gNn0KZm9yKHN0YXQgaW4gc3RhdHNfb3ZlcmFsbCkgewogICAgc3RhdF91bml0cyA8LSB1bml0cyR1bml0c1t1bml0cyRtZXRyaWM9PXN0YXRdCiAgICBpZihzdGF0X3VuaXRzPT0iIikgewogICAgICAgIGNhcCA8LSBzdGF0CiAgICB9IGVsc2UgewogICAgICAgIGNhcCA8LSBwYXN0ZTAoc3RhdCwgIiwgIiwgc3RhdF91bml0cykKICAgIH0KICAgIGNhdChwYXN0ZTAoIjxoND4iLGNhcCwiPC9oND5cbiIpKQogICAgcGVyZiA8LSBocGNjICU+JSBmaWx0ZXIobWV0cmljID09IHN0YXQpICU+JQogICAgICAgIGdyb3VwX2J5KHJlc291cmNlLGV4ZV90eXBlLGNwdXMpICU+JQogICAgICAgIHN1bW1hcmlzZShhdmVyYWdlPW1lYW4odmFsdWUpLHN0ZGV2PXNkKHZhbHVlKSkgJT4lCiAgICAgICAgdW5ncm91cCgpICU+JSBtdXRhdGUoY29uZj1wYXN0ZShyZXNvdXJjZSxleGVfdHlwZSxjcHVzLCBzZXAgPSAiLCAiKSkgJT4lCiAgICAgICAgbXV0YXRlKGNvbmY9ZmFjdG9yKGNvbmYsIGxldmVscyA9IHJldihjb25mKSkpCiAgICBwcmludChnZ3Bsb3QocGVyZiwgYWVzKHg9Y29uZiwgeT1hdmVyYWdlKSkgKwogICAgICAgIGdlb21fYmFyKHN0YXQ9ImlkZW50aXR5IiwgZmlsbD0icm95YWxibHVlIiwgY29sb3I9InJveWFsYmx1ZTQiKSArCiAgICAgICAgZ2VvbV9lcnJvcmJhcihhZXMoeW1pbj1hdmVyYWdlLXN0ZGV2LCB5bWF4PWF2ZXJhZ2Urc3RkZXYpLCB3aWR0aD0uMikrCiAgICAgICAgY29vcmRfZmxpcCgpK3lsYWIoY2FwKSkKICAgIHByaW50KGthYmxlKHBlcmYgJT4lIHNlbGVjdCgtY29uZiksIGNhcHRpb24gPSBjYXAsIGRpZ2l0cyA9IDIsICJodG1sIikgJT4lCiAgICAgICAga2FibGVfc3R5bGluZyhmdWxsX3dpZHRoID0gRikpCn0KZm9yKHN0YXQgaW4gc3RhdHNfcGVyX2NvcmUpIHsKICAgIHN0YXRfdW5pdHMgPC0gdW5pdHMkdW5pdHNbdW5pdHMkbWV0cmljPT1zdGF0XQogICAgaWYoc3RhdF91bml0cz09IiIpIHsKICAgICAgICBjYXAgPC0gc3RhdAogICAgfSBlbHNlIHsKICAgICAgICBjYXAgPC0gcGFzdGUwKHN0YXQsICIsICIsIHN0YXRfdW5pdHMpCiAgICB9CiAgICBjYXQocGFzdGUwKCI8aDQ+IixjYXAsIjwvaDQ+XG4iKSkKICAgIHBlcmYgPC0gaHBjYyAlPiUgZmlsdGVyKG1ldHJpYyA9PSBzdGF0KSAlPiUKICAgICAgICBncm91cF9ieShyZXNvdXJjZSxleGVfdHlwZSxjcHVzKSAlPiUKICAgICAgICBzdW1tYXJpc2UoYXZlcmFnZT1tZWFuKHZhbHVlKmNwdXMpLHN0ZGV2PXNkKHZhbHVlKmNwdXMpKSAlPiUKICAgICAgICB1bmdyb3VwKCkgJT4lIG11dGF0ZShjb25mPXBhc3RlKHJlc291cmNlLGV4ZV90eXBlLGNwdXMsIHNlcCA9ICIsICIpKSAlPiUKICAgICAgICBtdXRhdGUoY29uZj1mYWN0b3IoY29uZiwgbGV2ZWxzID0gcmV2KGNvbmYpKSkKICAgIHByaW50KGdncGxvdChwZXJmLCBhZXMoeD1jb25mLCB5PWF2ZXJhZ2UpKSArCiAgICAgICAgZ2VvbV9iYXIoc3RhdD0iaWRlbnRpdHkiLCBmaWxsPSJyb3lhbGJsdWUiLCBjb2xvcj0icm95YWxibHVlNCIpICsKICAgICAgICBnZW9tX2Vycm9yYmFyKGFlcyh5bWluPWF2ZXJhZ2Utc3RkZXYsIHltYXg9YXZlcmFnZStzdGRldiksIHdpZHRoPS4yKSsKICAgICAgICBjb29yZF9mbGlwKCkreWxhYihjYXApKQogICAgcHJpbnQoa2FibGUocGVyZiAlPiUgc2VsZWN0KC1jb25mKSwgY2FwdGlvbiA9IGNhcCwgZGlnaXRzID0gMiwgImh0bWwiKSAlPiUKICAgICAgICBrYWJsZV9zdHlsaW5nKGZ1bGxfd2lkdGggPSBGKSkKfQpgYGAKCiMjIyBQZXJmb3JtYW5jZSBwZXIgQ29yZSAodGhhdCBpcyBNUEkgUHJvY2VzcykKCmBgYHtyIHJlc3VsdHM9J2FzaXMnLCBlY2hvPUZBTFNFfQpmb3Ioc3RhdCBpbiBzdGF0c19vdmVyYWxsKSB7CiAgICBzdGF0X3VuaXRzIDwtIHVuaXRzJHVuaXRzW3VuaXRzJG1ldHJpYz09c3RhdF0KICAgIGlmKHN0YXRfdW5pdHM9PSIiKSB7CiAgICAgICAgY2FwIDwtIHN0YXQKICAgIH0gZWxzZSB7CiAgICAgICAgY2FwIDwtIHBhc3RlMChzdGF0LCAiLCAiLCBzdGF0X3VuaXRzKQogICAgfQogICAgY2F0KHBhc3RlMCgiPGg0PiIsY2FwLCI8L2g0PlxuIikpCiAgICBwZXJmIDwtIGhwY2MgJT4lIGZpbHRlcihtZXRyaWMgPT0gc3RhdCkgJT4lCiAgICAgICAgZ3JvdXBfYnkocmVzb3VyY2UsZXhlX3R5cGUsY3B1cykgJT4lCiAgICAgICAgc3VtbWFyaXNlKGF2ZXJhZ2U9bWVhbih2YWx1ZS9jcHVzKSxzdGRldj1zZCh2YWx1ZSkpICU+JQogICAgICAgIHVuZ3JvdXAoKSAlPiUgbXV0YXRlKGNvbmY9cGFzdGUocmVzb3VyY2UsZXhlX3R5cGUsY3B1cywgc2VwID0gIiwgIikpICU+JQogICAgICAgIG11dGF0ZShjb25mPWZhY3Rvcihjb25mLCBsZXZlbHMgPSByZXYoY29uZikpKQogICAgcHJpbnQoZ2dwbG90KHBlcmYsIGFlcyh4PWNvbmYsIHk9YXZlcmFnZSkpICsKICAgICAgICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIsIGZpbGw9InJveWFsYmx1ZSIsIGNvbG9yPSJyb3lhbGJsdWU0IikgKwogICAgICAgIGdlb21fZXJyb3JiYXIoYWVzKHltaW49YXZlcmFnZS1zdGRldiwgeW1heD1hdmVyYWdlK3N0ZGV2KSwgd2lkdGg9LjIpKwogICAgICAgIGNvb3JkX2ZsaXAoKSt5bGFiKGNhcCkpCiAgICBwcmludChrYWJsZShwZXJmICU+JSBzZWxlY3QoLWNvbmYpLCBjYXB0aW9uID0gY2FwLCBkaWdpdHMgPSAyLCAiaHRtbCIpICU+JQogICAgICAgIGthYmxlX3N0eWxpbmcoZnVsbF93aWR0aCA9IEYpKQp9CmZvcihzdGF0IGluIHN0YXRzX3Blcl9jb3JlKSB7CiAgICBzdGF0X3VuaXRzIDwtIHVuaXRzJHVuaXRzW3VuaXRzJG1ldHJpYz09c3RhdF0KICAgIGlmKHN0YXRfdW5pdHM9PSIiKSB7CiAgICAgICAgY2FwIDwtIHN0YXQKICAgIH0gZWxzZSB7CiAgICAgICAgY2FwIDwtIHBhc3RlMChzdGF0LCAiLCAiLCBzdGF0X3VuaXRzKQogICAgfQogICAgY2F0KHBhc3RlMCgiPGg0PiIsY2FwLCI8L2g0PlxuIikpCiAgICBwZXJmIDwtIGhwY2MgJT4lIGZpbHRlcihtZXRyaWMgPT0gc3RhdCkgJT4lCiAgICAgICAgZ3JvdXBfYnkocmVzb3VyY2UsZXhlX3R5cGUsY3B1cykgJT4lCiAgICAgICAgc3VtbWFyaXNlKGF2ZXJhZ2U9bWVhbih2YWx1ZSksc3RkZXY9c2QodmFsdWUqY3B1cykpICU+JQogICAgICAgIHVuZ3JvdXAoKSAlPiUgbXV0YXRlKGNvbmY9cGFzdGUocmVzb3VyY2UsZXhlX3R5cGUsY3B1cywgc2VwID0gIiwgIikpICU+JQogICAgICAgIG11dGF0ZShjb25mPWZhY3Rvcihjb25mLCBsZXZlbHMgPSByZXYoY29uZikpKQogICAgcHJpbnQoZ2dwbG90KHBlcmYsIGFlcyh4PWNvbmYsIHk9YXZlcmFnZSkpICsKICAgICAgICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIsIGZpbGw9InJveWFsYmx1ZSIsIGNvbG9yPSJyb3lhbGJsdWU0IikgKwogICAgICAgIGdlb21fZXJyb3JiYXIoYWVzKHltaW49YXZlcmFnZS1zdGRldiwgeW1heD1hdmVyYWdlK3N0ZGV2KSwgd2lkdGg9LjIpKwogICAgICAgIGNvb3JkX2ZsaXAoKSt5bGFiKGNhcCkpCiAgICBwcmludChrYWJsZShwZXJmICU+JSBzZWxlY3QoLWNvbmYpLCBjYXB0aW9uID0gY2FwLCBkaWdpdHMgPSAyLCAiaHRtbCIpICU+JQogICAgICAgIGthYmxlX3N0eWxpbmcoZnVsbF93aWR0aCA9IEYpKQp9CmBgYAo=